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= Positive Risk Balance (PRB): 
e Claim of risk lower than a human driver 
e But, how can you be sure this is true? 


= Positive Trust Balance: 


e Claim that you can trust predicted PRB 


e Four components: 
— Validation 
— Good engineering 
— Field feedback 
— Strong safety culture 


> Test it Right 

> Build it Right 

=> Improve it Right 
=> Live it Right 
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Positive Risk Balance (PRB) 


= PRB =» safer than human drivers 
e 1x?, 10x? or 100x? (other metric?) 
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= Brute force: drive a few billion miles 
e How often does safety driver intervene? 
e Invalidated if operational domain changes 
e Changing software resets odometer 
... too expensive, takes too long 





= Practical approach requires simulation 
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Simulation Validity 





= All parts have to be valid to get a valid prediction 


Autonomy 
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Hypothetical Validation Campaign (@ sescancu 
= 10,000M mile simulation campaign 
e Goal: under 1 fatality/billion miles 
e Claim ~10x PRB if simulation is valid 
= 100M mile collected data/scenarios 
e Claim simulating this is representative 
= 10M road testing of final software 
e Claim this validates simulation 





= Is this statistically valid PRB? 
e Questionable confidence in collected data tig og 
e Road testing useful, but insufficient on its own 
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How Much Do You Trust Simulation? CG Rescarcr 


= Would put a child in front of a PRB self driving car? 
e 10,000M mile sims 
.. perhaps with a simulator error? 


e 100M miles data collected 
.. perhaps with scenario analysis errors? 


e 10M of road testing 
.. that missed the above errors? 


e Built from software binaries ~s 
.. With no safety analysis? (QO 


e With biased perception training data? 
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Claimed PRB Isn't Enough 


= Validation-only “PRB’ claim is really: 
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“We have PRB as far as we know” 





BUT: 
e Maybe we just got lucky that validation missed defects 
e Maybe we missed something in our models 
e Maybe we had confirmation bias due to time pressure 


= Where's the safety argument? 
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; hacichacbochnosich 
= Stakeholders must trust that system is safe enough 

e Validation predicts PRB 


e Trust that PRB estimate is as valid as you can make it 
e Trust continuous improvement based on experience 


Positive Trust Balance 


Validations., 
Engineering Rigor Trustworthy 
PRB — 

Prediction 





Field Engineering Feedback 


safety Culture 
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= Testing alone is insufficient for life-critical systems 
e So we use also use engineering rigor 


Engineering Rigor 


= Can you trust the system itself? 
e Is it engineered for safety? 
e Were standards and best practices used? 
e Is there a safety case documenting all this? 


= Can you trust your validation process? 
e Did you engineer the simulations properly? 
e Did you design the validation campaign properly? 
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Field Engineering Feedback 


= Expected risk has a mean + uncertainty 
e You should deploy only when PRB mean is acceptable 
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e But, there will be uncertainty 
— Missed edge cases during road testing 
— Unknown gaps in validation plan 
— Unknown unknowns in general 
= Solution: continuous field monitoring 
e Monitor Safety Performance Indicators (SPIs) 
— SPI violation means safety argument has a defect 
— Investigate and fix root causes before loss events 
e Start during validation; continue after deployment 
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Safety Culture 


= Did you do what you said you did? 
e Did your validation skip over known problems? 
e Did your engineering team skip process steps? 
e Is your field monitoring ignoring SPI violations? 


= Good safety culture mitigates risk 
e Having a Safety Management System is a start 
e Safety culture involves everyone in the lifecycle 








https://bit.ly/3i5wl57 


= Safety culture simplified: 
e Are you incentivized to do the right thing? 
e Is it OK to tell your boss bad news? Will your boss fix a J 


Positive Trust Balance 


= Positive Trust Balance: 
e Stakeholders trust that lifecycle risk will be acceptable (e.g., PRB) 


TRUSTWORTHY POSITIVE RISK BALANCE 
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Engineering Validation Feedback Safety 
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